{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extract LAT Data\n",
    "\n",
    "This thread shows how to extract LAT data from the FERMI Science Support Center (FSSC) [archive](http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi) and perform further selection cuts using the Fermitools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synopsis:\n",
    "\n",
    "This thread leads you through extracting your LAT data files from the FSSC's databases. In order to analyze LAT data, you will need several [LAT data products](https://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_DP.html).\n",
    "\n",
    "1. An events file containing the recorded events that correspond to your source of interest, as well as the area surrounding that source. The size of this surrounding \"region-of-interest\", or ROI, that you choose will depend on the density and brightness of nearby sources, as well as the type of analysis you are performing. There are two types of events files:\n",
    "    * **Photon Data** - contains all information necessary for science analysis with `source`, `clean`, `ultraclean`, `ultracleanveto`, or `sourceveto` event classes\n",
    "    * **Extended Data** - contains all event data, including the standard _transient_ classes (`TRANSIENT010` and `TRANSIENT020`), plus additional quantities produced by the Level 1 analysis\n",
    "\n",
    "   Only one type of event data file is needed to perform LAT science analysis.\n",
    "   \n",
    "   We recommend use of the photon data file for most purposes (except for the [GRB analysis](https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/explore_latdata_burst.html), for which the extended data are required). Event data for large datasets will be divided into multiple files. The next tutorial, [Data Preparation](https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/data_preparation.html), will discuss how to combine those files.\n",
    "   \n",
    "   \n",
    "2. A spacecraft file containing spacecraft position and orientation information at 30 second intervals. This file is required for LAT science analysis. The LAT data server will only generate a single spacecraft file, regardless of the size of the dataset.\n",
    "\n",
    "\n",
    "3. Some analyses also require models of the isotropic and Galactic diffuse background models. You can download them from the [Background Models](https://fermi.gsfc.nasa.gov/ssc/data/access/lat/BackgroundModels.html) page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A description of the contents of each file is available [here](https://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html).\n",
    "\n",
    ">**Please Note**: If you wish to download data for the full sky or a substantial fraction of it, please download and use the [LAT All-Sky Weekly files](http://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/LAT_weekly_allsky.html) for your analysis. The data server does not support queries with radii larger than 60 degrees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Procedure\n",
    "\n",
    "To select all photons within a circular region around the source:\n",
    "1. Go to the [FSSC's web site data server](http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi)\n",
    "2. In this thread we will use this set of parameters:\n",
    "\n",
    "    -  Object name or coordinates: 193.98, -5.82\n",
    "    -  Coordinate system: \"J2000\"\n",
    "    -  Search radius (degrees): 20\n",
    "    -  Observation dates: START, 255398400\n",
    "    -  Time system: \"MET\"\n",
    "    -  Energy range (MeV): 100, 500000\n",
    "    -  LAT data type: \"Photon\"\n",
    "    -  Spacecraft data: \"checked\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image,HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HTML(\"<iframe src='http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi' width='1000' height='500'></iframe>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/images/extract_data/extract_LAT_data.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Notes:\n",
    "\n",
    "All the following information can be retrieved at the [Help for Fermi Database Query Form](http://fermi.gsfc.nasa.gov/ssc/LATDataQuery_help.html) page. Here we present a shorter version of the help page:\n",
    "\n",
    "* The coordinates (whether Equatorial or Galactic) have to be in degrees and separated by a comma. You will need to save these coordinates for subsequent use in several of the Fermitools. The source coordinates will be reported back to you when you retrieve the results of your query. You can insert also the name (like \"Vela pulsar\") of the source and it will be resolved (i.e., the resolver will retrieve the source coordinates for you) using [SIMBAD](http://simbad.u-strasbg.fr/simbad/), [NED](http://ned.ipac.caltech.edu/), and [HEASARC GRB catalog (GRBCAT)](http://heasarc.gsfc.nasa.gov/W3Browse/gamma-ray-bursts/grbcat.html), name resolution services, in that order.\n",
    "\n",
    "\n",
    "* If a pair of coordinates are entered, you have to choose the equinox time (\"J2000\" or \"B1950\") if these were Celestial coordinates, or you have to select \"Galactic\" if these were Galactic coordinates. This selection is ignored if a name was given.\n",
    "\n",
    "\n",
    "* If no search radius is provided, the default value of 15 degrees will be used.\n",
    "\n",
    "\n",
    "* To select all photons in this region from a particular time range, enter the start and stop times, separated by a comma, in the Observations Dates field. Observation dates may be specified in the Gregorian calendar system, or as a number of seconds in the MET system (Mission Elapsed Time, floating-point number of seconds since midnight of 1 January 2001), or as a number of days in the MJD system (Modified Julian Days, floating-point number of days since midnight of 17 November, 1858). The [xTime](http://heasarc.gsfc.nasa.gov/cgi-bin/Tools/xTime/xTime.pl) utility is available should you need to convert your time to a different format.\n",
    "\n",
    "    Gregorian dates are specified using the format: YYYY-MM-DD HH:MM:SS (all values except the year must use a leading 0 to create a 2-character field, e.g., 2010-01-01 01:23:45).\n",
    "\n",
    "    The starting date may be replaced by the string 'START' (as in this example), to indicate the earliest available data. The ending date may be replaced by the string 'END', to indicate the most recent available data. The 'START' and 'END' keywords are case-insensitive.\n",
    "\n",
    "    If this field is not specified, the latest 6 months of data (based on the most recent photon data) will be used as the search range.\n",
    "\n",
    "    \"START\" and \"END\" can be used with any time format.\n",
    "\n",
    "    **Note**: In the current version of the query software, floating-point seconds for the \"Gregorian\" format are not supported (i.e., the seconds have to be rounded to an integer).\n",
    "    \n",
    "\n",
    "* Select the correct time system. The choice is between \"Gregorian\" (default selection), \"MJD\", and \"MET\".\n",
    "\n",
    "\n",
    "* To select the energy range, use two values in MeV separated by a comma (they are treated as the minimum and maximum photon or event energies). If no energy range is specified, the default energy range of 100 MeV to 300 GeV is used.\n",
    "\n",
    "\n",
    "* Select which LAT data type (if any) you want to download. The default option is \"Photon\" and is recommended for science. Extended data files are also available, which include all event classes (including the `transient` classes needed for Gamma-ray Burst analysis) and provide additional information for each event. Extended data files can replace the photon data in your analysis. One can retrieve the \"Spacecraft\" data file without downloading the photon or event data file: In this case, choose \"None\" for this option.\n",
    "\n",
    "\n",
    "* The Spacecraft data selection is enabled by default. If you do not wish to download the \"Spacecraft\" data file, remember to uncheck this selection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Click on the **Start Search** button.\n",
    "\n",
    "The 'Query Submitted' webpage will be displayed and provide an estimate of the time to complete the query, as well as a link to the results webpage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/images/extract_data/query_LAT_submitted.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you go to this new webpage —'LAT Data Query Results' — you may be told that the query is not yet complete. Periodically, click on your browser's refresh button. When the query is complete, the Position in Queue will show as 'Query Complete' and the data file list will include links to the files themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/images/extract_data/results_query.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. It is strongly suggested that you capture the information reported on the query page in a text file accompanying your data. While it can be retrieved through other means, having the information from your query easily accesible may ease certain portions of the analysis.\n",
    "\n",
    "\n",
    "5. Download the spacecraft (pointing and livetime history) file and events data file to your working directory. There may be multiple events files (`_PH##` or `_EV##`), but there should be only a single spacecraft (`_SC##`) file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/images/extract_data/results_links.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, the results page includes a list of `wget` commands that you can copy and paste into a terminal window to retrieve the files on the command line.\n",
    "\n",
    "If `wget` does not work for you, consider `curl <link to download> -o <filename>`.\n",
    "\n",
    "6. Next, you should prepare your data for analysis by following the [Data Preparation Tutorial](https://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/data_preparation.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
